NASA S P-6002 


PROGRAM STANDARDS 






OFFICE OF MANNED SPACE FLIGHT 


RELIABILITY PROGRAM EVALUATION PROCEDURES 

(R-l) 


PREFACE 


Reliability Program Evaluation Procedures and related survey checklists are 
established as a standard to assure consistent evaluations of reliability proce- 
dures and controls being applied to Manned Space Flight Programs. More 
specifically, the objectives are threefold: 

a. To establish uniform standards for evaluating the degree and 
effectiveness of reliability practices and controls. 

b. To identify reliability type problems for evaluation and correction. 

c. To permit evaluation of the various methods of controlling a specific 
area leading to improved reliability and safety levels. 

This standard is based on and is consistent with NASA Publication NPC 250-1; 
however, it may be used to survey contractual compliance to all reliability pub- 
lications. It is designed to identify problem and improvement areas consistent 
with the severe reliability and safety requirements of Maimed Space Flight Systems. 

Comments and questions concerning the requirements set forth in this publication 
should be referred to the Office of Manned Space Flight (Code MIR-D), NASA 
Headquarters, Washington 25, D.C. Questions concerning its application to 
specific contracts should be referred to the cognizant NASA Center. 
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1 . 


EVALUATION PROCEDURES 


1 . 1 INTRODUCTION 

The reliability and quality requirements of the Manned Space Flight Program demand 
design, manufacture, test and operations reliability, and quality levels that far exceed 
those required in the past. The Reliability Program Evaluation Procedures presented 
in this section have been developed to assist in assessing the degree that reliability 
programs are properly related to the over-all needs of the Manned Space Flight Pro- 
gram, and to provide a base from which the specific action required to make the 
programs more effective can be developed. 

1 . 2 RESPONSIBILITIES 

1.2.1 OMSF Responsibilities 

The Office of Manned Space Flight will: 

a. Be responsible for the establishment and revision of evaluation 
procedures . 

b. Insure follow-up review of the reliability program requirements to 
determine the actions taken on deviations noted during the evaluation. 

c. Monitor scheduling and maintain a record of all evaluations and 
follow-up reviews. 

1.2.2 NASA Center Responsibilities 

The cognizant NASA Centers will be responsible for implementing an effective pro- 
gram of periodic reliability program evaluations (surveys). More specifically, this 
will involve the following responsibilities: 

a. Schedule surveys . 

b. Designate a chairman of and direct the survey team. 

c. Notify the contractor, OMSF, and if applicable, the cognizant Government 
representative by letter at least 30 days prior to the date of the proposed 
survey. 

d. Conduct a presurvey conference at which team members will meet with 
contractor personnel at the contractor’s facility. The survey team 
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chairman will explain the objectives and general plan of the evaluation 
for the understanding of all concerned. 

e. Conduct the surveys and evaluate the reliability program utilizing the 
procedures and checklists outlined herein. 

f. Conduct a postsurvey critique with contractor personnel and discuss the 
preliminary results of the evaluation. The contractor should be given an 
opportunity to explain any unusual or discrepant information obtained. 

g. The chairman of the survey team will be responsible for preparing a 
final report of the evaluation for the NASA Center and OMSF. Copies of 
this report will be sent to evaluation team members and other activities 
as necessary. The contractor will be notified, in writing, of the results 
of the evaluation and action necessary to correct or improve deficiencies . 

h. Follow up specific survey results to determine the action taken as a result 
of deficiencies noted during the survey. The assistance of the cognizant 
Government representative and resident Apollo System Project Office, if 
applicable, will normally be utilized to the maximum extent in this 
follow-up. 

i. Maintain records of all survey reports and related follow-up summaries. 

j . Transmit to OMSF copies of all survey reports and related follow-up 
summaries with copies to other NASA Centers and team members as 
appropriate. 

1.2.3 Survey Representatives 

Survey teams will usually be comprised of the following members: 

a. NASA Center chairman and designated representatives. 

b. Cognizant Government representative (if applicable). 

c. OMSF representatives. 

1 . 3 ACTIVITY AREAS 

Reliability program activities consist essentially of a network of interrelated proce- 
dures and controls that are designed to assure an end product which meets Manned 
Space Flight Program needs. This reliability program extends throughout the entire 
organization in its Work Element coverage; and in its time-phased coverage extends 
from initial contract definition throughout the entire program. 
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A reliability program can be considered to consist of 12 major Activity Areas, each 
bearing a separate and distinct relationship to the over-all program. These Activity 
Areas are described in NPC 250-1 and are listed as follows: 

1. Program Management. 

2. Design Specifications. 

3. Reliability Prediction and Estimation. 

4. Failure Mode, Effect, and Criticality Analysis. 

5. Human Engineering and Maintainability. 

6. Design Review Program. 

7. Failure Reporting and Correction. 

8. Standardization of Design Practices. 

9. Parts and Materials Program. 

10. Equipment Logs. 

11. Reliability Evaluation. 

12. Documentation of Reliability Program. 

An evaluation of the Degree of Effective Coverage for each Activity Area can be estab- 
lished by determining the importance of individual Work Elements and the determina- 
tion of the Degree of Effective Coverage provided for each within individual Activity 
Areas. Similarly an over-all Reliability Program Evaluation of an entire reliability 
program can be developed from compiling the results of the individual Activity Area 
evaluation. 

1.4 EVALUATION PROCEDURE 
1.4.1 Objectives 

Reliability evaluations are conducted to provide a means of: 

a. Determining effective reliability program coverage in terms of Manned 
Space Flight Program needs. 

b. Determining effective reliability program coverage in terms of specific 
contractual requirements . 

c. Determining the relative strengths and weaknesses in each of the major 
Activity Areas . 

d. Determining the relative strengths and weaknesses of the individual Work 
Elements that make up each Activity Area. 
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e. Measuring, through subsequent surveys, changes in effectiveness of 
reliability activities. 

f. Making recommendations for improving, strengthening, or de-emphasizing 
Activity Areas. 

1.4.2 Evaluation Procedure Steps 

In meeting the above objectives the Reliability Program Evaluation is performed in the 
following basic steps . 

a. Determining the Relative Importance (in percent) of each of the reliability 
Activity Areas to the specific program in terms of Manned Space Flight 
Program needs . 

b. Determining the Relative Importance (1 to 10) of the individual Work 
Elements within each of the Activity Areas . 

c. Establishing the Degree of Effective Coverage (0, 20, 40, 60, 80, or 
100 percent) of the individual Work Elements within each of the 
Activity Areas . 

d. Listing related document number and date (where applicable) for 
individual Work Elements. 

e. Determining the current assignment of Functional Responsibility for 
each of the individual Work Elements . 

f . Developing a Weighted Effective Rating for each Work Element by 
multiplying the Relative Importance Factor by the Degree of Effective 
Coverage. 

g. For each Work Element subtracting the Weighted Effective Coverage 
Rating from the Relative Importance Factor to evaluate each Work 
Element in terms of need for action and priority. (The higher the 
number the greater the need.) 

h. Developing (similar to step f) a composite Weighted Effective Coverage 
Rating for each Activity Area based on Effective Coverage Ratings on 
individual Work Elements. 

i. For each Activity Area (similar to step g) subtracting the Weighted 
Effective Coverage Rating from the Relative Importance Factor to 
evaluate each Activity Area in terms of need for action and priority. 

(The higher the number the greater the need.) 

j. Developing and establishing specific recommendations to increase the 
effectiveness of the Manned Space Flight Reliability Program. 
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k. Reviewing survey reports to identify Activity Areas and Work Elements 
where improved reliability procedures and controls are needed. 

l. Reviewing survey reports to determine Activity Areas and Work Elements 
where exceptionally effective reliability procedures and controls have been 
identified. 

1.4.3 Contractual Compliance Procedure Steps 

In determining compliance to specific contractual requirements, only a slight modifica- 
tion to the above procedure is required and is accomplished as follows: 

a. The individual Work Elements within the Activity Areas are recollated, 
as applicable, against the requirements of the specific contractual docu- 
ments, thereby replacing the Activity Areas by Contractual Require- 
ment Areas. 

b. Establishment of Relative Importance Factors for the Work Elements 
within the Contractual Requirement Areas. 

c. Upon completion of step e, in the preceding 12-step procedure, the 
results of step c of that procedure (Degree of Effective Coverage), 
step d (applicable document number and date), and step e (assignment 
of Functional Responsibility) are posted to this revised breakdown. 

d. Steps f through 1 are then repeated on a Contractual Requirement Area 
basis instead of an Activity Area basis. 

Contractual requirement evaluations will supplement, not replace, the procedure of 
paragraph 1.4.2 due to the greater coverage required by increased reliability and 
safety needs of Manned Space Flight Programs. 

1.5 SCORING METHODS 

1.5.1 Determining the Relative Importance of Activity Areas 

Each of the Activity Areas listed in paragraph 1 . 3 has a separate and distinct contri- 
bution in a reliability program. However, all of these Activity Areas are not independ- 
ent and a major weakness within a reliability program in one of these Activity Areas 
can have a decided effect upon the contribution of the other Activity Areas . 

In the implementation of the Reliability Program Survey, the first step is to establish 
Relative Importance Factors for each Activity Area. For initial planning purposes, a 
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set of Relative Importance Factors has been established as shown on page 2-5. These 
Relative Importance Factors can be revised in later surveys to reflect adjustments 
indicated for the type of program being surveyed. 

1.5.2 Determining the Relative Importance of the Individual Work Elements of 
Each Activity Area 

Each of the Activity Areas is made up of a number of Work Elements . These elements 
describe the key reliability procedures and controls that are necessary to obtain maxi- 
mum results from the Activity Area. The Work Elements are in the form of numbered 
declarative statements with an affirmative response scoring favorably. The number of 
Work Elements used for each activity varies with the complexity of the activity. The 
Work Elements are grouped under subheadings for easy reference . It is intended that 
the Work Elements, but not the subheadings, be scored. These Work Elements can be 
used in a survey regardless of the specifications and/or other requirements of the con- 
tract as indicated in paragraph 1.4.3. Work Elements within an Activity Area are not 
equally important. Initial Relative Importance Factors have been established for each 
Work Element as noted in Column A of the survey checklist (Section 2) . These factors 
will be subject to revision based on survey experience. 

1.5.3 Rating Work Elements for Degree of Effective Coverage 

The successful application of a Reliability Program Survey in fulfilling its objectives 
as a program status and improvement tool, lies in the logical and accurate evaluation 
of the Degree of Effective Coverage currently provided against each Work Element. 

It should be recognized that the assignment of the Relative Importance Factor for Work 
Elements (paragraph 1.5.2) and the determination of their current Degree of Effective 
Coverage are directed at establishing those Work Elements and those activities that 
merit the highest priority of action to strengthen the program coverage and effective- 
ness. Some Work Elements of an Activity Area may be highly important but have a 
low Degree of Effective Coverage. It is those Work Elements that have the highest 
combination of Relative Importance and lack of Effective Coverage that will merit 
highest priority of action. 

As shown in Column B of Figure 1-1, when rating the Degree of Effective Coverage, 
each Work Element will be given a rating of 0, 20, 40, 60, 80, or 100 percent. This 
rating expresses, in percentage points, the Degree of Effective Coverage of the Work 
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Element found by the survey team personnel. Satisfactory coverage warrants a rating 
of 100 percent. Ratings less than 100 percent should be supported by adequate notes. 

The product of Column B and Column A is now inserted in Column C to give a quantity 
that represents the Weighted Effective Coverage of the Work Element. 

1.5.4 Determining Document Numbers and Dates 

Where applicable to individual work statements, the related document numbers, titles, 
and dates which reflect compliance are recorded as indicated in Figure 1-1. This 
might be a special or periodic report or procedure, for example. 

1.5.5 Determining Current Assignment of Functional Responsibility for 
Work Elements 

The Column E (Function Responsible for Work Element) will be filled- in to indicate 
the organization unit responsible for the Work Element. 

The possibility of multiple assignments or the lack of assignments must also be 
recorded as appropriate . Important details that cannot be stated in this column 
should be supplied in a supporting narrative report, which will be referenced in 
Column E. 

1.5.6 Determining the Relative Need for Action of the Individual Work 
Elements of an Activity 

When reviewing the results of surveys or when recommending corrective action, it is 
desirable to point out the extent of lack of coverage of Work Elements and to include at 
the same time the Relative Importance aspect of the Work Elements. To do this, sub- 
tract Weighted Degree of Effective Coverage (Column C) from Established Importance 
Factor (Column A). The result is then placed in Column D. The higher the number, 
the greater the need. 

1.5.7 Developing Activity Area Effective Coverage Ratings 

The Degree of Effective Coverage of an Activity Area is a function of the Degree of 
Effective Coverage of its individual Work Elements weighted by their Importance Fac- 
tor. The mechanics of determining the Activity Area Degree of Effective Coverage 
are shown in Figure 1-2 and are detailed as follows: 
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a. Add the Weighted Degree of Effective Coverage for all Work Elements 
(Column C) to obtain a total for the activity. 

b. Add the Relative Importance Factors for all Work Elements (Column A) 
to obtain a total for the activity. 

c. Divide the activity total for Weighted Degree of Effective Coverage 
(step a) by the activity total for the Relative Importance Factor (step b) . 
The result is the Activity Degree of Effective Coverage and should be 
noted on the last page of the Activity Work Element sheets. As shown 
in Figure 1-3, the Activity Degree of Effective Coverage number should 
also be inserted in Column B of the reliability program evaluation 
summary sheet. 

1.5.8 Rating Activity Areas in Terms of Need for Action 

The procedure for rating Activity Areas in terms of need for action is identical to the 
procedure for rating the Work Elements. The Relative Need (Column D) is the dif- 
ference between the Established Importance Factor (Column A) and the Weighted 
Degree of Effective Coverage (Column C). 

1.5.9 Determining Over-all Degree of Effective Coverage 

The over-all Degree of Effective Coverage for each reliability program is established 
directly from the Relative Importance Factors of the individual Activity Areas and 
their associated degree of current Effective Coverage. It is developed in the same 
manner as the Degree of Effective Coverage of the Activity Areas was established 
except all Importance Factors must add to 100 percent as shown in Figure 1-3. 

The Relative Importance Factor (Column A) established for each Activity Area is 
multiplied by the Degree of Effective Coverage determined for that activity (Column B), 
and the result, representing a Weighted Effective Coverage figure in percent, is 
inserted in Column C. 

The sum of Weighted Effective Coverages (Column C) is then totaled and is a percent 
representing the degree of over-all reliability program coverage. 
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Figure 1-3. Example of Reliability Program Evaluation Summary Sheet Calculations 
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RELIABILITY PROGRAM EVALUATION CHECKLISTS 


2. 1 INTRODUCTION 

This section contains the Reliability Program Evaluation Summary Sheet for rating 
and evaluating the twelve Activity Areas. It also contains the individual Activity Area 
checklists for use in rating and evaluating the Work Elements that comprise each 
Activity Area. 
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New information acquired from similar projects. 
Improvements resulting from invention and innovation. 
Specific reliability engineering analyses. 
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